1 Load Packages

Let’s first load the packages that we need for this chapter.

1.1 Settings

1.2 Functions

2 Data

Data & analysis code was pulled from the ROAR-Score repository on Github (https://github.com/yeatmanlab/ROAR-Score/tree/main/Analysis/SRE_research/data).

2.1 Data Loading (Reference Dataframes)

2.2 Data Loading (Response Matrix)

3 Study 1: Creation and analysis of sentence reading efficiency items

We created a list of 200 sentences for Sentence Reading Efficiency, an online, automated assessment in which participants endorse sentences as being true or false. These sentences were created by human researchers who referenced the sentence structure of other sentence reading efficiency tasks such as the Test of Silent Reading Comprehension and Efficiency (TOSREC) and the Woodcock-Johnson Sentence Reading Fluency (WJ-SRF).

167 participants aged 5-29 who were recruited across 3 different studies piloted the task. Based on these data, we conducted item-level analysis to filter out items that are not suitable for the task, leaving us with 136 final sentences.

Setting variables that will be used in code later on

3.1 Participant Sample for Item Analysis

The assessment offers the participants only 2 choices: true or false; participants who are guessing can be expected to get ~50% of the items corrects. Thus, we conservatively remove participants from our sample if they could not correctly endorse more than 60% (0.60) of the items. We filtered out 16 participants, leaving us with 151 participants for the final sample.

3.1.1 Data wrangling for response matrices (Lab & TOSREC)

3.1.2 Data wrangling for RT matrices (Lab)

3.1.3 Filtering out participants who have less than 60% correct

df.filtered.participants <- df.resp.matrix.lab.filtered %>%
  mutate(keep = 1) %>%
  select(subj, keep)

3.2 Removing Items for Final Sentences

We filtered out items that had a proportion correct of 85% or greater for our initial pass of the stimuli bank.

We retained items that had a proportion correct greater than 0.85 (85%) for two reasons: 1) We wanted to ensure that the task would be relatively easy for participants, and 2) we need a sufficient number of sentences so participants are not able to exhaust the whole stimuli pool. We also filtered out items that had 25 or less responses.

After filtering out 38 items, we were left with 162 items that we conducted further analysis on.

3.3 Inspecting relationship between agreeability and average reaction time (RT)

quad.plot <- ggplot(data = df.lab.items.filtered.v3, label = sentence, aes(label = sentence)) + 
  geom_point(data = df.lab.items.filtered.v3, mapping = aes(x = avgRTStd,
                     y = pCorrStd, color=numCharacter)) + 
  labs(title = "Response Time vs. Agreement Rate",
       color = "Sentence Length \n(Number of \nCharacters)") + 
  scale_x_continuous(limits = c(-3, 3), breaks = round(seq(-3,3)), "Faster <-------   Response Time ----> Slower") +
  scale_y_continuous(limits = c(-3, 3), breaks = round(seq(-3,3)),"Less <-------   Agreeability   ---->  More ") +
  scale_color_viridis(option = 'viridis') +
  theme_light() +
  theme(text=element_text(family="Avenir", size=12)) 
ggplotly(quad.plot)

3.3.1 Removing a sentence because of the long response time despite the short character length

df.lab.items.filtered.v4 <- df.lab.items.filtered.v3 %>%
  filter(sentenceId != "lab_56")

3.3.2 Matching items based on easy, medium, and hard difficulty

In order to ensure that we have an equal number of true and false sentences, we used the R package MatchIt to match the items based on item difficulty (or proportion correct). Note that difficulties for each item were defined as:

  • Easy: Items that have a proportion correct greater than 0.95

  • Medium: Items that have a proportion correct greater than 0.90 but less than or equal to 0.95

  • Hard: Items that have a proportion correct less than than 0.90

3.3.3 Matching easy sentences

After matching the easy sentences, 40 sentences (20 true and 20 false) were included in the final set of sentences.

3.3.3.1 Excluded easy sentences

df.easy.excluded
   sentenceId                                                  sentence
1      lab_46                               Knives are used for cutting
2      lab_54                               Some kids have blue crayons
3      lab_79                              Chocolate is a type of candy
4      lab_83                            Some people keep birds as pets
5      lab_84                    Some people eat pancakes for breakfast
6     lab_135                    Breakfast is the first meal of the day
7     lab_138                   Some people get to work using the train
8     lab_158       Many people like wearing shoes when they go outside
9     lab_171 When it is raining outside, many fields will become muddy
10    lab_181        Some kids want to become writers when they grow up
   answerKey itemNum attempted correct     pCorr  pCorrStd       sum    avgRT
1       TRUE      46        31      30 0.9677419 0.8820929  52.38303 1.689775
2       TRUE      54        42      40 0.9523810 0.6655083 129.07389 3.073188
3       TRUE      79        35      34 0.9714286 0.9340733  63.32416 1.862475
4       TRUE      83        45      43 0.9555556 0.7102691 135.12499 3.002778
5       TRUE      84        42      40 0.9523810 0.6655083 181.61277 4.324114
6       TRUE     135        37      36 0.9729730 0.9558488  74.21376 2.005777
7       TRUE     138        35      34 0.9714286 0.9340733 138.04104 3.944030
8       TRUE     158        33      32 0.9696970 0.9096583 128.09033 3.881525
9       TRUE     171        41      39 0.9512195 0.6491324 158.71274 4.069557
10      TRUE     181        37      36 0.9729730 0.9558488 114.99220 3.194228
       avgRTStd numCharacter difficulty
1  -1.513551206           23       Easy
2  -0.125084880           23       Easy
3  -1.340220211           23       Easy
4  -0.195752505           25       Easy
5   1.130410378           33       Easy
6  -1.196394600           31       Easy
7   0.748938125           32       Easy
8   0.686204942           43       Easy
9   0.874924119           48       Easy
10 -0.003602929           41       Easy

3.3.3.2 Final easy sentences

df.lab.items.easy.final
   sentenceId
1       lab_0
2       lab_1
3       lab_4
4       lab_6
5       lab_8
6      lab_12
7      lab_16
8      lab_25
9      lab_32
10     lab_33
11     lab_35
12     lab_40
13     lab_44
16     lab_55
17     lab_61
18     lab_63
19     lab_69
20     lab_71
21     lab_73
22     lab_74
23     lab_75
24     lab_77
28     lab_85
29     lab_87
30     lab_89
31     lab_90
32     lab_93
33     lab_99
34    lab_106
35    lab_115
36    lab_116
37    lab_122
40    lab_140
42    lab_162
43    lab_164
44    lab_170
46    lab_178
47    lab_180
49    lab_186
50    lab_191
                                                               sentence
1                                                      Chairs are alive
2                                                      Fish have thumbs
3                                                      Cars have wheels
4                                                   Trees have branches
5                                                   Sandwiches are food
6                                                  Soda is always brown
7                                                   Children can be sad
8                                                   All cars are yellow
9                                             Triangles have five sides
10                                                  A monkey has a face
11                                               People like to eat mud
12                                     Children enjoy playing with toys
13                                   Computers need electricity to work
16                                          Some houses have back yards
17                                         Some kids take music lessons
18                                      Spaceships are flown by cowboys
19                                            A pillow can be very soft
20                                   Some worms ride pumpkins to school
21                                         Sweaters can be made of coal
22                                       Chefs cook meals in their beds
23                                         Dogs always walk on two legs
24                                     Trash is thrown away in bathtubs
28                                   Eyeglasses can help you see better
29                                          Wolves are a type of monkey
30                                          It is usually dark at night
31                                           A blanket can be very soft
32                                       Towels are often made of glass
33                                   Most humans listen with their feet
34                                 Some children are afraid of the dark
35                                People can buy tomatoes at the market
36                                   All children are afraid of the sun
37                                Orange juice is always made of grapes
40                            The ocean is home to many aquatic animals
42                  Trash cans are where you store extra notebook paper
43                       Basketball is a game that you play with a fork
44                  Some people like to run for exercise in the morning
46                       Soccer is a game you play with only your hands
47                         A teacher is a person that works at a school
49 Grandparents can bring presents when they come to visit their family
50               When you break a bone you normally have to wear a cast
   answerKey itemNum attempted correct     pCorr  pCorrStd       sum    avgRT
1      FALSE       0        44      42 0.9545455 0.6960271  91.49515 2.079435
2      FALSE       1        45      45 1.0000000 1.3369207  85.00037 1.888897
3       TRUE       4        42      41 0.9761905 1.0012145  66.08075 1.611726
4       TRUE       6        41      40 0.9756098 0.9930265  67.65853 1.650208
5       TRUE       8        48      47 0.9791667 1.0431778  73.21354 1.525282
6      FALSE      12        43      41 0.9534884 0.6811226  93.69506 2.178955
7       TRUE      16        41      40 0.9756098 0.9930265  66.20243 1.614694
8      FALSE      25        35      35 1.0000000 1.3369207  62.23937 1.778268
9      FALSE      32        40      40 1.0000000 1.3369207  93.43161 2.335790
10      TRUE      33        32      32 1.0000000 1.3369207  51.59473 1.612335
11     FALSE      35        47      45 0.9574468 0.7369352  89.03229 1.894304
12      TRUE      40        37      37 1.0000000 1.3369207  89.03335 2.406307
13      TRUE      44        29      29 1.0000000 1.3369207  75.53671 2.604714
16      TRUE      55        43      43 1.0000000 1.3369207  89.82713 2.089003
17      TRUE      61        39      39 1.0000000 1.3369207  88.94155 2.280552
18     FALSE      63        44      44 1.0000000 1.3369207 127.36084 2.894565
19      TRUE      69        48      47 0.9791667 1.0431778 105.97524 2.207817
20     FALSE      71        34      33 0.9705882 0.9222248 104.39870 3.070550
21     FALSE      73        41      39 0.9512195 0.6491324 125.96161 3.072235
22     FALSE      74        41      40 0.9756098 0.9930265 118.94683 2.973671
23     FALSE      75        42      41 0.9761905 1.0012145 123.97459 2.951776
24     FALSE      77        45      43 0.9555556 0.7102691 136.87046 3.110692
28      TRUE      85        38      37 0.9736842 0.9658770 108.14471 2.922830
29     FALSE      87        41      39 0.9512195 0.6491324  76.64538 1.965266
30      TRUE      89        38      38 1.0000000 1.3369207  85.07677 2.238862
31      TRUE      90        34      34 1.0000000 1.3369207 105.06983 3.090289
32     FALSE      93        43      43 1.0000000 1.3369207  93.89001 2.183489
33     FALSE      99        45      44 0.9777778 1.0235949 116.48522 2.588561
34      TRUE     106        37      37 1.0000000 1.3369207  82.42713 2.227760
35      TRUE     115        37      37 1.0000000 1.3369207 115.53913 3.122679
36     FALSE     116        40      39 0.9750000 0.9844292  96.30437 2.407609
37     FALSE     122        34      34 1.0000000 1.3369207  78.62520 2.312506
40      TRUE     140        38      37 0.9736842 0.9658770 128.94337 3.484956
42     FALSE     162        35      34 0.9714286 0.9340733 197.23792 5.635369
43     FALSE     164        45      45 1.0000000 1.3369207 135.32318 3.007182
44      TRUE     170        51      50 0.9803922 1.0604568 172.20617 3.376592
46     FALSE     178        35      34 0.9714286 0.9340733 122.21326 3.491807
47      TRUE     180        38      37 0.9736842 0.9658770  93.46986 2.459733
49      TRUE     186        38      37 0.9736842 0.9658770 165.01177 4.342415
50      TRUE     191        41      40 0.9756098 0.9930265 166.95486 4.072070
      avgRTStd numCharacter difficulty  distance weights subclass
1  -1.12246785           14       Easy 0.5585887       1        7
2  -1.31370199           14       Easy 0.6372482       1        3
3  -1.59188587           14       Easy 0.5966747       1        8
4  -1.55326285           17       Easy 0.5956658       1       13
5  -1.67864527           17       Easy 0.6018322       1       18
6  -1.02258443           17       Easy 0.5567070       1       14
7  -1.58890710           16       Easy 0.5956658       1       20
8  -1.42473554           16       Easy 0.6372482       1        2
9  -0.86517627           22       Easy 0.6372482       1        1
10 -1.59127394           15       Easy 0.6372482       1        1
11 -1.30827509           18       Easy 0.5637444       1       20
12 -0.79440217           28       Easy 0.6372482       1        2
13 -0.59527002           30       Easy 0.6372482       1        3
16 -1.11286499           23       Easy 0.6372482       1        4
17 -0.92061586           24       Easy 0.6372482       1        5
18 -0.30436081           27       Easy 0.6372482       1        4
19 -0.99361660           20       Easy 0.6018322       1        6
20 -0.12773249           29       Easy 0.5869091       1       13
21 -0.12604183           23       Easy 0.5526629       1       17
22 -0.22496551           25       Easy 0.5956658       1       15
23 -0.24694053           23       Easy 0.5966747       1       12
24 -0.08744370           27       Easy 0.5603851       1       19
28 -0.27599197           29       Easy 0.5923149       1        7
29 -1.23705399           22       Easy 0.5526629       1       16
30 -0.96245822           22       Easy 0.6372482       1        9
31 -0.10792127           21       Easy 0.6372482       1       10
32 -1.01803426           25       Easy 0.6372482       1        5
33 -0.61148261           29       Easy 0.5994281       1       11
34 -0.97360101           30       Easy 0.6372482       1       11
35 -0.07541288           31       Easy 0.6372482       1       12
36 -0.79309479           28       Easy 0.5946056       1       18
37 -0.88854555           31       Easy 0.6372482       1        9
40  0.28818725           34       Easy 0.5923149       1       14
42  2.44645576           43       Easy 0.5883785       1        8
43 -0.19133219           37       Easy 0.6372482       1       10
44  0.17942712           42       Easy 0.6039494       1       15
46  0.29506376           37       Easy 0.5883785       1        6
47 -0.74078061           35       Easy 0.5923149       1       16
49  1.14877841           58       Easy 0.5923149       1       17
50  0.87744587           43       Easy 0.5956658       1       19

3.3.4 Matching medium sentences

df.lab.items.medium <- df.lab.items.filtered.difficulty %>%
  filter(difficulty == "Medium")

After matching the medium sentences, 64 sentences (32 true and 32 false) were included in the final set of sentences.

3.3.4.1 Excluded medium sentences

df.medium.excluded
  sentenceId
1      lab_2
2     lab_62
3     lab_72
4    lab_103
5    lab_109
6    lab_132
7    lab_137
8    lab_175
9    lab_197
                                                                sentence
1                                                         Cats can drink
2                                       Squirrels will climb trees often
3                                           A baker works in the kitchen
4                                        People can buy chairs in stores
5                                    Some people put ketchup on hot dogs
6                           Some people drink coffee with milk and sugar
7                      Wearing sunscreen protects your skin from the sun
8                   Many people have the same last name as their parents
9 The president of the United States of America lives in the White House
  answerKey itemNum attempted correct     pCorr  pCorrStd       sum    avgRT
1      TRUE       2        37      35 0.9459459 0.5747769  81.23219 2.195465
2      TRUE      62        39      37 0.9487179 0.6138612  92.50852 2.372013
3      TRUE      72        36      34 0.9444444 0.5536063 113.64815 3.247090
4      TRUE     103        39      37 0.9487179 0.6138612 105.89910 2.786818
5      TRUE     109        36      34 0.9444444 0.5536063 108.85277 3.110079
6      TRUE     132        35      33 0.9428571 0.5312258 119.19430 3.505715
7      TRUE     137        37      35 0.9459459 0.5747769 118.15913 3.375975
8      TRUE     175        39      37 0.9487179 0.6138612 148.08023 3.896848
9      TRUE     197        38      36 0.9473684 0.5948333 107.93718 2.917221
     avgRTStd numCharacter difficulty
1 -1.00601452           12     Medium
2 -0.82882074           28     Medium
3  0.04945240           23     Medium
4 -0.41250052           26     Medium
5 -0.08805895           29     Medium
6  0.30902199           37     Medium
7  0.17880849           42     Medium
8  0.70158407           43     Medium
9 -0.28162165           58     Medium

3.3.4.2 Final medium sentences

df.lab.items.medium.final
   sentenceId                                                  sentence
2       lab_3                                            Bears can talk
3       lab_7                                            Mice have ears
4      lab_10                                         The sun is bright
5      lab_11                                      Whales have two legs
6      lab_13                                    All flowers are yellow
7      lab_14                               People sometimes drink milk
8      lab_15                                        Cows sing at night
9      lab_18                               Elephants are large animals
10     lab_21                                      Apples grow on trees
11     lab_22                                    Kittens are young cats
12     lab_27                                     Some shoes have laces
13     lab_28                                         Adults can be sad
14     lab_30                                     Moms like eating hats
15     lab_31                                    Many babies drink milk
16     lab_36                                Some kids love red crayons
17     lab_37                                Lizards like to cook pasta
18     lab_38                              Snails are usually very fast
19     lab_39                              Pets can keep people company
20     lab_45                                Blankets are made of metal
21     lab_49                                 Every magician is a fairy
22     lab_50                               Blankets help keep you cold
23     lab_51                             Many nurses work in hospitals
24     lab_53                                   People like to eat dust
25     lab_58                     Tigers sometimes swing at playgrounds
26     lab_60                                 Some girls play the piano
28     lab_64                               Cloudy days are always warm
29     lab_66                             People drive cars in the sink
30     lab_70                           A curtain is something you wear
32     lab_81                          Eating only ice cream is healthy
33     lab_88                             People drive cars on the road
34     lab_91                       Some slugs ride elephants to school
35     lab_95                     Puppies carry backpacks on their back
36     lab_98                              Most houses are made of foam
37    lab_100              Many people paint pictures with screwdrivers
38    lab_101                 Lollipops are always made from vegetables
40    lab_105                     Most houses have grass on the ceiling
41    lab_107                        Pockets can be used to keep change
43    lab_110                  People brush their teeth with mayonnaise
44    lab_114                           A cactus may grow in the desert
45    lab_119                  Boys usually wash their hair with yogurt
46    lab_120                  Girls can listen to music with earphones
47    lab_121                  Climbing up a mountain can be exhausting
48    lab_123             Adults are usually taller than their children
49    lab_124          Dentists always give children candy and lemonade
50    lab_127                            A picture can hang on the wall
51    lab_128                      People can use cash to pay in stores
53    lab_134                 The United States of America is in Africa
54    lab_136                        Towels are used to make things dry
56    lab_144                     Most girls keep their books in a pond
57    lab_145               Every basketball player is on a school team
58    lab_146                     Stores are where you go to buy things
59    lab_153             People can wear socks to keep their feet warm
60    lab_154      Air conditioners can lower the temperature of a room
61    lab_157         Medicine is supposed to make you stay sick longer
62    lab_165 Umbrellas are useful for keeping people dry when it rains
63    lab_166     Spiders like to drink orange juice out of coffee mugs
64    lab_167             Baseball is a game that you play with a knife
65    lab_169              When a traffic light is red you need to stop
67    lab_176            Most dogs wear leashes when they go for a walk
68    lab_179                    It is faster to run than it is to walk
69    lab_182    Students never get in trouble when they cheat on tests
70    lab_183   A doctor can work in a hospital that serves many people
71    lab_184    Some people like to read books before they go to sleep
72    lab_193    Friends can cheer you up when you are having a bad day
   answerKey itemNum attempted correct     pCorr    pCorrStd       sum    avgRT
2      FALSE       3        41      37 0.9024390 -0.03865587  56.47353 1.411838
3       TRUE       7        54      50 0.9259259  0.29250144 129.53265 2.444012
4       TRUE      10        34      32 0.9411765  0.50752893  44.29897 1.384343
5      FALSE      11        46      43 0.9347826  0.41737766 141.54039 3.145342
6      FALSE      13        50      47 0.9400000  0.49094110 100.29683 2.046874
7       TRUE      14        46      42 0.9130435  0.11086332 116.87339 2.597187
8      FALSE      15        34      32 0.9411765  0.50752893  58.45122 1.719154
9       TRUE      18        43      39 0.9069767  0.02532443  87.28487 2.128899
10      TRUE      21        46      42 0.9130435  0.11086332  77.55469 1.723438
11      TRUE      22        35      32 0.9142857  0.12837842  76.54983 2.251466
12      TRUE      27        34      31 0.9117647  0.09283306 100.60881 3.048752
13      TRUE      28        33      30 0.9090909  0.05513344  60.19951 1.824228
14     FALSE      30        37      34 0.9189189  0.19370503  86.52711 2.403531
15      TRUE      31        41      38 0.9268293  0.30523826  83.98658 2.153502
16      TRUE      36        44      40 0.9090909  0.05513344 128.42074 2.986529
17     FALSE      37        33      31 0.9393939  0.48239585  93.09233 2.820980
18     FALSE      38        38      35 0.9210526  0.22378965  86.69107 2.281344
19      TRUE      39        44      40 0.9090909  0.05513344 126.34444 3.008201
20     FALSE      45        42      38 0.9047619 -0.00590405 106.11772 2.652943
21     FALSE      49        38      35 0.9210526  0.22378965 110.77468 2.915123
22     FALSE      50        41      37 0.9024390 -0.03865587 107.19094 2.679773
23      TRUE      51        35      32 0.9142857  0.12837842  79.38833 2.268238
24     FALSE      53        42      39 0.9285714  0.32980213  73.74313 1.798613
25     FALSE      58        42      39 0.9285714  0.32980213 135.62898 3.308024
26      TRUE      60        36      33 0.9166667  0.16194904  81.75420 2.270950
28     FALSE      64        34      31 0.9117647  0.09283306 130.60357 3.957684
29     FALSE      66        33      30 0.9090909  0.05513344  74.02196 2.387805
30     FALSE      70        34      32 0.9411765  0.50752893 103.98862 3.058489
32     FALSE      81        40      38 0.9500000  0.63193769 108.47540 2.711885
33      TRUE      88        37      34 0.9189189  0.19370503  74.58953 2.071932
34     FALSE      91        47      43 0.9148936  0.13694964 164.55758 3.501225
35     FALSE      95        47      44 0.9361702  0.43694240 117.58834 2.501880
36     FALSE      98        35      33 0.9428571  0.53122584  96.19204 2.829178
37     FALSE     100        36      34 0.9444444  0.55360625 136.59557 3.794321
38     FALSE     101        42      39 0.9285714  0.32980213 150.71802 3.767951
40     FALSE     105        52      48 0.9230769  0.25233147 185.66851 3.640559
41      TRUE     107        42      38 0.9047619 -0.00590405 150.20310 3.663490
43     FALSE     110        38      36 0.9473684  0.59483333 142.59378 3.853886
44      TRUE     114        45      42 0.9333333  0.39694337 106.92999 2.430227
45     FALSE     119        35      33 0.9428571  0.53122584  91.81476 2.782266
46      TRUE     120        39      36 0.9230769  0.25233147 128.24418 3.288312
47      TRUE     121        34      31 0.9117647  0.09283306 119.06712 3.501974
48      TRUE     123        49      45 0.9183673  0.18592805 148.56686 3.031977
49     FALSE     124        40      37 0.9250000  0.27944620 135.08062 3.463606
50      TRUE     127        33      31 0.9393939  0.48239585  88.90841 2.778388
51      TRUE     128        47      44 0.9361702  0.43694240 184.63177 3.928336
53     FALSE     134        37      35 0.9459459  0.57477691 117.87427 3.185791
54      TRUE     136        48      44 0.9166667  0.16194904 234.40883 4.883517
56     FALSE     144        42      39 0.9285714  0.32980213 143.57695 3.589424
57     FALSE     145        34      31 0.9117647  0.09283306 124.27937 3.766042
58      TRUE     146        34      31 0.9117647  0.09283306 125.22210 3.794609
59      TRUE     153        33      31 0.9393939  0.48239585 112.60909 3.519034
60      TRUE     154        32      30 0.9375000  0.45569195 140.25530 4.524365
61     FALSE     157        46      43 0.9347826  0.41737766 169.29087 3.936997
62      TRUE     165        39      36 0.9230769  0.25233147 161.19087 4.241865
63     FALSE     166        32      30 0.9375000  0.45569195 125.63388 4.052706
64     FALSE     167        37      35 0.9459459  0.57477691  98.14500 2.804143
65      TRUE     169        49      45 0.9183673  0.18592805 132.50265 2.760472
67      TRUE     176        43      40 0.9302326  0.35322349 138.08943 3.211382
68      TRUE     179        33      31 0.9393939  0.48239585 124.76787 3.898996
69     FALSE     182        44      40 0.9090909  0.05513344 178.93396 4.066681
70      TRUE     183        43      40 0.9302326  0.35322349 145.54502 3.465358
71      TRUE     184        49      45 0.9183673  0.18592805 177.13930 3.690402
72      TRUE     193        42      38 0.9047619 -0.00590405 117.03945 2.786653
      avgRTStd numCharacter difficulty  distance weights subclass
2  -1.79250328           12     Medium 0.6233724       1       31
3  -0.75655902           12     Medium 0.5656060       1       10
4  -1.82009928           14     Medium 0.5270103       1       12
5  -0.05266739           17     Medium 0.5432581       1       11
6  -1.15514793           19     Medium 0.5300052       1        5
7  -0.60282510           24     Medium 0.5976186       1       28
8  -1.48406552           15     Medium 0.5270103       1       29
9  -1.07282289           24     Medium 0.6124280       1       32
10 -1.47976573           17     Medium 0.5976186       1        1
11 -0.94980896           19     Medium 0.5945631       1        2
12 -0.14961015           18     Medium 0.6007560       1        3
13 -1.37860749           14     Medium 0.6072892       1        4
14 -0.79718809           18     Medium 0.5831041       1       16
15 -1.04813035           19     Medium 0.5633372       1        5
16 -0.21206070           22     Medium 0.6072892       1        6
17 -0.37821431           22     Medium 0.5315473       1       10
18 -0.91982146           24     Medium 0.5777957       1        1
19 -0.19030930           24     Medium 0.6072892       1        7
20 -0.54686498           22     Medium 0.6177849       1       13
21 -0.28372710           21     Medium 0.5777957       1       28
22 -0.51993660           23     Medium 0.6233724       1       21
23 -0.93297543           25     Medium 0.5945631       1        8
24 -1.40431574           19     Medium 0.5589543       1       25
25  0.11060888           33     Medium 0.5589543       1       17
26 -0.93025338           21     Medium 0.5886865       1        9
28  0.76264208           23     Medium 0.6007560       1        7
29 -0.81297151           24     Medium 0.6072892       1        4
30 -0.13983788           26     Medium 0.5270103       1       26
32 -0.48770755           27     Medium 0.5045013       1       12
33 -1.12999885           24     Medium 0.5831041       1       11
34  0.30451595           30     Medium 0.5930652       1        3
35 -0.69848013           32     Medium 0.5397386       1       15
36 -0.36998656           23     Medium 0.5227286       1       19
37  0.59868278           39     Medium 0.5186816       1       23
38  0.57221569           36     Medium 0.5589543       1       20
40  0.44435863           31     Medium 0.5727429       1        2
41  0.46737360           28     Medium 0.6177849       1       13
43  0.65846492           35     Medium 0.5112206       1       27
44 -0.77039439           25     Medium 0.5469294       1       14
45 -0.41706997           34     Medium 0.5227286       1       14
46  0.09082529           34     Medium 0.5727429       1       15
47  0.30526759           34     Medium 0.6007560       1       16
48 -0.16644666           39     Medium 0.5844732       1       17
49  0.26675882           42     Medium 0.5679286       1        8
50 -0.42096193           24     Medium 0.5315473       1       18
51  0.73318658           29     Medium 0.5397386       1       19
53 -0.01207044           34     Medium 0.5148511       1       22
54  1.69185734           28     Medium 0.5886865       1       20
56  0.39303667           30     Medium 0.5589543       1        9
57  0.57029960           36     Medium 0.6007560       1        6
58  0.59897154           30     Medium 0.6007560       1       21
59  0.32239001           37     Medium 0.5315473       1       22
60  1.33139271           44     Medium 0.5363620       1       23
61  0.74187955           41     Medium 0.5432581       1       30
62  1.04786111           48     Medium 0.5727429       1       24
63  0.85801106           44     Medium 0.5363620       1       24
64 -0.39511266           36     Medium 0.5148511       1       18
65 -0.43894331           35     Medium 0.5844732       1       25
67  0.01361405           37     Medium 0.5547667       1       26
68  0.70373970           29     Medium 0.5315473       1       27
69  0.87203733           45     Medium 0.6072892       1       32
70  0.26851733           45     Medium 0.5547667       1       29
71  0.49438387           44     Medium 0.5844732       1       30
72 -0.41266609           43     Medium 0.6177849       1       31

3.3.5 Matching hard sentences

After matching the hard sentences, 26 sentences (13 true and 13 false) were included in the final set of sentences.

3.3.5.1 Excluded hard sentences

df.hard.excluded
  sentenceId                                                sentence answerKey
1     lab_20                                Fire gives you frostbite     FALSE
2     lab_26                                  All girls wear dresses     FALSE
3     lab_59                         Camels enjoy wearing flip-flops     FALSE
4     lab_67                           Stones are kept in the fridge     FALSE
5     lab_97                          Every fish lives in a fishbowl     FALSE
6    lab_108                       The sun is smaller than the Earth     FALSE
7    lab_125                      Every door is a sliding glass door     FALSE
8    lab_130                Leaves on trees turn green when they die     FALSE
9    lab_173 Most adults walk with their hands instead of their feet     FALSE
  itemNum attempted correct     pCorr   pCorrStd       sum    avgRT    avgRTStd
1      20        42      37 0.8809524 -0.3416102  94.69809 2.367452 -0.83339861
2      26        44      39 0.8863636 -0.2653134  99.65924 2.264983 -0.93624256
3      59        36      31 0.8611111 -0.6213654  85.44953 2.513221 -0.68709690
4      67        37      32 0.8648649 -0.5684387  71.62725 2.046493 -1.15553045
5      97        36      32 0.8888889 -0.2297082 126.69766 3.519379  0.32273649
6     108        37      33 0.8918919 -0.1873669  82.42029 2.354865 -0.84603155
7     125        37      33 0.8918919 -0.1873669 117.72194 3.270054  0.07250017
8     130        46      41 0.8913043 -0.1956510 188.94052 4.198678  1.00451670
9     173        37      32 0.8648649 -0.5684387 141.11462 3.919851  0.72467064
  numCharacter difficulty
1           21       Hard
2           19       Hard
3           28       Hard
4           24       Hard
5           25       Hard
6           27       Hard
7           28       Hard
8           33       Hard
9           46       Hard

3.3.5.2 Final hard sentences

df.lab.items.hard.final
   sentenceId                                                     sentence
1       lab_9                                             Wheels are round
2      lab_17                                        Doors are always open
5      lab_34                                            A raven is a bird
6      lab_41                               Dish washers make plates dirty
7      lab_48                                    A tricycle has two wheels
8      lab_52                           Trampolines help you bounce higher
10     lab_68                                 A cactus grows under the sea
11     lab_76                              There are many types of lizards
12     lab_80                      Television shows are longer than movies
13     lab_86                                Ice helps to keep things cold
14     lab_92                                 A gardener works at a bakery
15     lab_96                              Snow always falls in the summer
17    lab_102                           Halloween is a holiday in February
19    lab_111                             A mouse is a rodent that squeaks
20    lab_113                             Stones can be found near a river
21    lab_117                            Hats should be worn on your elbow
24    lab_131               Every child holds pencils with their left hand
25    lab_133                          Most babies know how to drive a car
26    lab_139                    Some people get to work using a spaceship
27    lab_142            Police officers drive school busses while on duty
28    lab_148             Friends can do many things together after school
29    lab_149                         Another name for a present is a gift
30    lab_152               The weather in the winter can be unpredictable
31    lab_156                 People can use credit cards to pay in stores
33    lab_185 Trashcans can become smelly if trash is left inside too long
34    lab_187  Most people play board games with at least one other person
   answerKey itemNum attempted correct     pCorr   pCorrStd       sum    avgRT
1       TRUE       9        40      35 0.8750000 -0.4255368  66.24548 1.698602
2      FALSE      17        36      31 0.8611111 -0.6213654  84.30579 2.408737
5       TRUE      34        46      40 0.8695652 -0.5021654  84.24562 1.831426
6      FALSE      41        41      36 0.8780488 -0.3825500 133.94117 3.266858
7      FALSE      48        39      34 0.8717949 -0.4707280 116.55780 2.988662
8       TRUE      52        35      31 0.8857143 -0.2744690  89.75446 2.639837
10     FALSE      68        47      41 0.8723404 -0.4630359 108.47005 2.410446
11      TRUE      76        38      34 0.8947368 -0.1472540 110.11218 2.897689
12     FALSE      80        41      36 0.8780488 -0.3825500 147.65833 3.691458
13      TRUE      86        41      36 0.8780488 -0.3825500 101.78939 2.482668
14     FALSE      92        38      34 0.8947368 -0.1472540  94.15842 2.544822
15     FALSE      96        38      34 0.8947368 -0.1472540 148.46899 4.012675
17     FALSE     102        42      37 0.8809524 -0.3416102 117.20619 2.790623
19      TRUE     111        42      37 0.8809524 -0.3416102 199.60910 4.752598
20      TRUE     113        38      34 0.8947368 -0.1472540 154.91296 4.186837
21     FALSE     117        35      31 0.8857143 -0.2744690 109.86183 3.329146
24     FALSE     131        33      29 0.8787879 -0.3721290 105.47607 3.196244
25     FALSE     133        42      37 0.8809524 -0.3416102 133.25655 3.250160
26     FALSE     139        27      23 0.8518519 -0.7519178 134.02384 4.963846
27     FALSE     142        39      35 0.8974359 -0.1091983 196.32806 5.306164
28      TRUE     148        39      35 0.8974359 -0.1091983 159.18083 4.081560
29      TRUE     149        38      33 0.8684211 -0.5182977 210.40058 5.536857
30      TRUE     152        40      35 0.8750000 -0.4255368 213.40882 5.335221
31      TRUE     156        32      28 0.8750000 -0.4255368  95.05799 3.066387
33      TRUE     185        34      29 0.8529412 -0.7365587 127.02520 3.969537
34      TRUE     187        35      30 0.8571429 -0.6773164 171.10255 5.032428
       avgRTStd numCharacter difficulty  distance weights subclass
1  -1.504691943           14       Hard 0.4001698       1        1
2  -0.791963193           18       Hard 0.4663906       1       11
5  -1.371382512           13       Hard 0.4257874       1       12
6   0.069292288           26       Hard 0.3860251       1        8
7  -0.209920087           21       Hard 0.4152211       1        7
8  -0.560018946           30       Hard 0.3513447       1       13
10 -0.790248253           23       Hard 0.4126471       1       12
11 -0.301225127           26       Hard 0.3124676       1        2
12  0.495443891           34       Hard 0.3860251       1        1
13 -0.717762058           24       Hard 0.3860251       1        3
14 -0.655380914           23       Hard 0.3124676       1        2
15  0.817834431           26       Hard 0.3124676       1        5
17 -0.408681593           29       Hard 0.3727302       1        4
19  1.560459528           26       Hard 0.3727302       1        4
20  0.992631846           26       Hard 0.3124676       1        5
21  0.131808505           27       Hard 0.3513447       1       13
24 -0.001578946           39       Hard 0.3826239       1        9
25  0.052533487           28       Hard 0.3727302       1        3
26  1.772479511           34       Hard 0.5113574       1       10
27  2.116047832           42       Hard 0.3013030       1        6
28  0.886970399           41       Hard 0.3013030       1        6
29  2.347584124           29       Hard 0.4312367       1        7
30  2.145210816           39       Hard 0.4001698       1        8
31 -0.131910927           36       Hard 0.4001698       1        9
33  0.774538946           50       Hard 0.5060627       1       10
34  1.841311904           49       Hard 0.4856379       1       11

3.3.6 Final set of sentences

We are left with a final stimuli bank of 130 sentences, consisting of 40 easy, 64 medium, and 26 hard sentences.

df.lab.items.final <- df.lab.items.easy.final %>%
  rbind(df.lab.items.medium.final) %>%
  rbind(df.lab.items.hard.final) 
df.lab.items.final
    sentenceId
1        lab_0
2        lab_1
3        lab_4
4        lab_6
5        lab_8
6       lab_12
7       lab_16
8       lab_25
9       lab_32
10      lab_33
11      lab_35
12      lab_40
13      lab_44
16      lab_55
17      lab_61
18      lab_63
19      lab_69
20      lab_71
21      lab_73
22      lab_74
23      lab_75
24      lab_77
28      lab_85
29      lab_87
30      lab_89
31      lab_90
32      lab_93
33      lab_99
34     lab_106
35     lab_115
36     lab_116
37     lab_122
40     lab_140
42     lab_162
43     lab_164
44     lab_170
46     lab_178
47     lab_180
49     lab_186
50     lab_191
27       lab_3
39       lab_7
410     lab_10
52      lab_11
66      lab_13
73      lab_14
81      lab_15
91      lab_18
101     lab_21
111     lab_22
121     lab_27
131     lab_28
14      lab_30
15      lab_31
161     lab_36
171     lab_37
181     lab_38
191     lab_39
201     lab_45
211     lab_49
221     lab_50
231     lab_51
241     lab_53
25      lab_58
26      lab_60
281     lab_64
291     lab_66
301     lab_70
321     lab_81
331     lab_88
341     lab_91
351     lab_95
361     lab_98
371    lab_100
38     lab_101
401    lab_105
41     lab_107
431    lab_110
441    lab_114
45     lab_119
461    lab_120
471    lab_121
48     lab_123
491    lab_124
501    lab_127
51     lab_128
53     lab_134
54     lab_136
56     lab_144
57     lab_145
58     lab_146
59     lab_153
60     lab_154
61     lab_157
62     lab_165
63     lab_166
64     lab_167
65     lab_169
67     lab_176
68     lab_179
69     lab_182
70     lab_183
71     lab_184
72     lab_193
110      lab_9
210     lab_17
55      lab_34
610     lab_41
74      lab_48
82      lab_52
102     lab_68
112     lab_76
122     lab_80
132     lab_86
141     lab_92
151     lab_96
172    lab_102
192    lab_111
202    lab_113
212    lab_117
242    lab_131
251    lab_133
261    lab_139
271    lab_142
282    lab_148
292    lab_149
302    lab_152
311    lab_156
332    lab_185
342    lab_187
                                                                sentence
1                                                       Chairs are alive
2                                                       Fish have thumbs
3                                                       Cars have wheels
4                                                    Trees have branches
5                                                    Sandwiches are food
6                                                   Soda is always brown
7                                                    Children can be sad
8                                                    All cars are yellow
9                                              Triangles have five sides
10                                                   A monkey has a face
11                                                People like to eat mud
12                                      Children enjoy playing with toys
13                                    Computers need electricity to work
16                                           Some houses have back yards
17                                          Some kids take music lessons
18                                       Spaceships are flown by cowboys
19                                             A pillow can be very soft
20                                    Some worms ride pumpkins to school
21                                          Sweaters can be made of coal
22                                        Chefs cook meals in their beds
23                                          Dogs always walk on two legs
24                                      Trash is thrown away in bathtubs
28                                    Eyeglasses can help you see better
29                                           Wolves are a type of monkey
30                                           It is usually dark at night
31                                            A blanket can be very soft
32                                        Towels are often made of glass
33                                    Most humans listen with their feet
34                                  Some children are afraid of the dark
35                                 People can buy tomatoes at the market
36                                    All children are afraid of the sun
37                                 Orange juice is always made of grapes
40                             The ocean is home to many aquatic animals
42                   Trash cans are where you store extra notebook paper
43                        Basketball is a game that you play with a fork
44                   Some people like to run for exercise in the morning
46                        Soccer is a game you play with only your hands
47                          A teacher is a person that works at a school
49  Grandparents can bring presents when they come to visit their family
50                When you break a bone you normally have to wear a cast
27                                                        Bears can talk
39                                                        Mice have ears
410                                                    The sun is bright
52                                                  Whales have two legs
66                                                All flowers are yellow
73                                           People sometimes drink milk
81                                                    Cows sing at night
91                                           Elephants are large animals
101                                                 Apples grow on trees
111                                               Kittens are young cats
121                                                Some shoes have laces
131                                                    Adults can be sad
14                                                 Moms like eating hats
15                                                Many babies drink milk
161                                           Some kids love red crayons
171                                           Lizards like to cook pasta
181                                         Snails are usually very fast
191                                         Pets can keep people company
201                                           Blankets are made of metal
211                                            Every magician is a fairy
221                                          Blankets help keep you cold
231                                        Many nurses work in hospitals
241                                              People like to eat dust
25                                 Tigers sometimes swing at playgrounds
26                                             Some girls play the piano
281                                          Cloudy days are always warm
291                                        People drive cars in the sink
301                                      A curtain is something you wear
321                                     Eating only ice cream is healthy
331                                        People drive cars on the road
341                                  Some slugs ride elephants to school
351                                Puppies carry backpacks on their back
361                                         Most houses are made of foam
371                         Many people paint pictures with screwdrivers
38                             Lollipops are always made from vegetables
401                                Most houses have grass on the ceiling
41                                    Pockets can be used to keep change
431                             People brush their teeth with mayonnaise
441                                      A cactus may grow in the desert
45                              Boys usually wash their hair with yogurt
461                             Girls can listen to music with earphones
471                             Climbing up a mountain can be exhausting
48                         Adults are usually taller than their children
491                     Dentists always give children candy and lemonade
501                                       A picture can hang on the wall
51                                  People can use cash to pay in stores
53                             The United States of America is in Africa
54                                    Towels are used to make things dry
56                                 Most girls keep their books in a pond
57                           Every basketball player is on a school team
58                                 Stores are where you go to buy things
59                         People can wear socks to keep their feet warm
60                  Air conditioners can lower the temperature of a room
61                     Medicine is supposed to make you stay sick longer
62             Umbrellas are useful for keeping people dry when it rains
63                 Spiders like to drink orange juice out of coffee mugs
64                         Baseball is a game that you play with a knife
65                          When a traffic light is red you need to stop
67                        Most dogs wear leashes when they go for a walk
68                                It is faster to run than it is to walk
69                Students never get in trouble when they cheat on tests
70               A doctor can work in a hospital that serves many people
71                Some people like to read books before they go to sleep
72                Friends can cheer you up when you are having a bad day
110                                                     Wheels are round
210                                                Doors are always open
55                                                     A raven is a bird
610                                       Dish washers make plates dirty
74                                             A tricycle has two wheels
82                                    Trampolines help you bounce higher
102                                         A cactus grows under the sea
112                                      There are many types of lizards
122                              Television shows are longer than movies
132                                        Ice helps to keep things cold
141                                         A gardener works at a bakery
151                                      Snow always falls in the summer
172                                   Halloween is a holiday in February
192                                     A mouse is a rodent that squeaks
202                                     Stones can be found near a river
212                                    Hats should be worn on your elbow
242                       Every child holds pencils with their left hand
251                                  Most babies know how to drive a car
261                            Some people get to work using a spaceship
271                    Police officers drive school busses while on duty
282                     Friends can do many things together after school
292                                 Another name for a present is a gift
302                       The weather in the winter can be unpredictable
311                         People can use credit cards to pay in stores
332         Trashcans can become smelly if trash is left inside too long
342          Most people play board games with at least one other person
    answerKey itemNum attempted correct     pCorr    pCorrStd       sum
1       FALSE       0        44      42 0.9545455  0.69602706  91.49515
2       FALSE       1        45      45 1.0000000  1.33692067  85.00037
3        TRUE       4        42      41 0.9761905  1.00121449  66.08075
4        TRUE       6        41      40 0.9756098  0.99302654  67.65853
5        TRUE       8        48      47 0.9791667  1.04317777  73.21354
6       FALSE      12        43      41 0.9534884  0.68112255  93.69506
7        TRUE      16        41      40 0.9756098  0.99302654  66.20243
8       FALSE      25        35      35 1.0000000  1.33692067  62.23937
9       FALSE      32        40      40 1.0000000  1.33692067  93.43161
10       TRUE      33        32      32 1.0000000  1.33692067  51.59473
11      FALSE      35        47      45 0.9574468  0.73693516  89.03229
12       TRUE      40        37      37 1.0000000  1.33692067  89.03335
13       TRUE      44        29      29 1.0000000  1.33692067  75.53671
16       TRUE      55        43      43 1.0000000  1.33692067  89.82713
17       TRUE      61        39      39 1.0000000  1.33692067  88.94155
18      FALSE      63        44      44 1.0000000  1.33692067 127.36084
19       TRUE      69        48      47 0.9791667  1.04317777 105.97524
20      FALSE      71        34      33 0.9705882  0.92222480 104.39870
21      FALSE      73        41      39 0.9512195  0.64913240 125.96161
22      FALSE      74        41      40 0.9756098  0.99302654 118.94683
23      FALSE      75        42      41 0.9761905  1.00121449 123.97459
24      FALSE      77        45      43 0.9555556  0.71026914 136.87046
28       TRUE      85        38      37 0.9736842  0.96587700 108.14471
29      FALSE      87        41      39 0.9512195  0.64913240  76.64538
30       TRUE      89        38      38 1.0000000  1.33692067  85.07677
31       TRUE      90        34      34 1.0000000  1.33692067 105.06983
32      FALSE      93        43      43 1.0000000  1.33692067  93.89001
33      FALSE      99        45      44 0.9777778  1.02359491 116.48522
34       TRUE     106        37      37 1.0000000  1.33692067  82.42713
35       TRUE     115        37      37 1.0000000  1.33692067 115.53913
36      FALSE     116        40      39 0.9750000  0.98442918  96.30437
37      FALSE     122        34      34 1.0000000  1.33692067  78.62520
40       TRUE     140        38      37 0.9736842  0.96587700 128.94337
42      FALSE     162        35      34 0.9714286  0.93407326 197.23792
43      FALSE     164        45      45 1.0000000  1.33692067 135.32318
44       TRUE     170        51      50 0.9803922  1.06045676 172.20617
46      FALSE     178        35      34 0.9714286  0.93407326 122.21326
47       TRUE     180        38      37 0.9736842  0.96587700  93.46986
49       TRUE     186        38      37 0.9736842  0.96587700 165.01177
50       TRUE     191        41      40 0.9756098  0.99302654 166.95486
27      FALSE       3        41      37 0.9024390 -0.03865587  56.47353
39       TRUE       7        54      50 0.9259259  0.29250144 129.53265
410      TRUE      10        34      32 0.9411765  0.50752893  44.29897
52      FALSE      11        46      43 0.9347826  0.41737766 141.54039
66      FALSE      13        50      47 0.9400000  0.49094110 100.29683
73       TRUE      14        46      42 0.9130435  0.11086332 116.87339
81      FALSE      15        34      32 0.9411765  0.50752893  58.45122
91       TRUE      18        43      39 0.9069767  0.02532443  87.28487
101      TRUE      21        46      42 0.9130435  0.11086332  77.55469
111      TRUE      22        35      32 0.9142857  0.12837842  76.54983
121      TRUE      27        34      31 0.9117647  0.09283306 100.60881
131      TRUE      28        33      30 0.9090909  0.05513344  60.19951
14      FALSE      30        37      34 0.9189189  0.19370503  86.52711
15       TRUE      31        41      38 0.9268293  0.30523826  83.98658
161      TRUE      36        44      40 0.9090909  0.05513344 128.42074
171     FALSE      37        33      31 0.9393939  0.48239585  93.09233
181     FALSE      38        38      35 0.9210526  0.22378965  86.69107
191      TRUE      39        44      40 0.9090909  0.05513344 126.34444
201     FALSE      45        42      38 0.9047619 -0.00590405 106.11772
211     FALSE      49        38      35 0.9210526  0.22378965 110.77468
221     FALSE      50        41      37 0.9024390 -0.03865587 107.19094
231      TRUE      51        35      32 0.9142857  0.12837842  79.38833
241     FALSE      53        42      39 0.9285714  0.32980213  73.74313
25      FALSE      58        42      39 0.9285714  0.32980213 135.62898
26       TRUE      60        36      33 0.9166667  0.16194904  81.75420
281     FALSE      64        34      31 0.9117647  0.09283306 130.60357
291     FALSE      66        33      30 0.9090909  0.05513344  74.02196
301     FALSE      70        34      32 0.9411765  0.50752893 103.98862
321     FALSE      81        40      38 0.9500000  0.63193769 108.47540
331      TRUE      88        37      34 0.9189189  0.19370503  74.58953
341     FALSE      91        47      43 0.9148936  0.13694964 164.55758
351     FALSE      95        47      44 0.9361702  0.43694240 117.58834
361     FALSE      98        35      33 0.9428571  0.53122584  96.19204
371     FALSE     100        36      34 0.9444444  0.55360625 136.59557
38      FALSE     101        42      39 0.9285714  0.32980213 150.71802
401     FALSE     105        52      48 0.9230769  0.25233147 185.66851
41       TRUE     107        42      38 0.9047619 -0.00590405 150.20310
431     FALSE     110        38      36 0.9473684  0.59483333 142.59378
441      TRUE     114        45      42 0.9333333  0.39694337 106.92999
45      FALSE     119        35      33 0.9428571  0.53122584  91.81476
461      TRUE     120        39      36 0.9230769  0.25233147 128.24418
471      TRUE     121        34      31 0.9117647  0.09283306 119.06712
48       TRUE     123        49      45 0.9183673  0.18592805 148.56686
491     FALSE     124        40      37 0.9250000  0.27944620 135.08062
501      TRUE     127        33      31 0.9393939  0.48239585  88.90841
51       TRUE     128        47      44 0.9361702  0.43694240 184.63177
53      FALSE     134        37      35 0.9459459  0.57477691 117.87427
54       TRUE     136        48      44 0.9166667  0.16194904 234.40883
56      FALSE     144        42      39 0.9285714  0.32980213 143.57695
57      FALSE     145        34      31 0.9117647  0.09283306 124.27937
58       TRUE     146        34      31 0.9117647  0.09283306 125.22210
59       TRUE     153        33      31 0.9393939  0.48239585 112.60909
60       TRUE     154        32      30 0.9375000  0.45569195 140.25530
61      FALSE     157        46      43 0.9347826  0.41737766 169.29087
62       TRUE     165        39      36 0.9230769  0.25233147 161.19087
63      FALSE     166        32      30 0.9375000  0.45569195 125.63388
64      FALSE     167        37      35 0.9459459  0.57477691  98.14500
65       TRUE     169        49      45 0.9183673  0.18592805 132.50265
67       TRUE     176        43      40 0.9302326  0.35322349 138.08943
68       TRUE     179        33      31 0.9393939  0.48239585 124.76787
69      FALSE     182        44      40 0.9090909  0.05513344 178.93396
70       TRUE     183        43      40 0.9302326  0.35322349 145.54502
71       TRUE     184        49      45 0.9183673  0.18592805 177.13930
72       TRUE     193        42      38 0.9047619 -0.00590405 117.03945
110      TRUE       9        40      35 0.8750000 -0.42553678  66.24548
210     FALSE      17        36      31 0.8611111 -0.62136538  84.30579
55       TRUE      34        46      40 0.8695652 -0.50216536  84.24562
610     FALSE      41        41      36 0.8780488 -0.38255001 133.94117
74      FALSE      48        39      34 0.8717949 -0.47072799 116.55780
82       TRUE      52        35      31 0.8857143 -0.27446899  89.75446
102     FALSE      68        47      41 0.8723404 -0.46303587 108.47005
112      TRUE      76        38      34 0.8947368 -0.14725402 110.11218
122     FALSE      80        41      36 0.8780488 -0.38255001 147.65833
132      TRUE      86        41      36 0.8780488 -0.38255001 101.78939
141     FALSE      92        38      34 0.8947368 -0.14725402  94.15842
151     FALSE      96        38      34 0.8947368 -0.14725402 148.46899
172     FALSE     102        42      37 0.8809524 -0.34161023 117.20619
192      TRUE     111        42      37 0.8809524 -0.34161023 199.60910
202      TRUE     113        38      34 0.8947368 -0.14725402 154.91296
212     FALSE     117        35      31 0.8857143 -0.27446899 109.86183
242     FALSE     131        33      29 0.8787879 -0.37212897 105.47607
251     FALSE     133        42      37 0.8809524 -0.34161023 133.25655
261     FALSE     139        27      23 0.8518519 -0.75191779 134.02384
271     FALSE     142        39      35 0.8974359 -0.10919826 196.32806
282      TRUE     148        39      35 0.8974359 -0.10919826 159.18083
292      TRUE     149        38      33 0.8684211 -0.51829769 210.40058
302      TRUE     152        40      35 0.8750000 -0.42553678 213.40882
311      TRUE     156        32      28 0.8750000 -0.42553678  95.05799
332      TRUE     185        34      29 0.8529412 -0.73655868 127.02520
342      TRUE     187        35      30 0.8571429 -0.67731641 171.10255
       avgRT     avgRTStd numCharacter difficulty  distance weights subclass
1   2.079435 -1.122467853           14       Easy 0.5585887       1    1_1_7
2   1.888897 -1.313701989           14       Easy 0.6372482       1    1_1_3
3   1.611726 -1.591885867           14       Easy 0.5966747       1    1_1_8
4   1.650208 -1.553262854           17       Easy 0.5956658       1   1_1_13
5   1.525282 -1.678645270           17       Easy 0.6018322       1   1_1_18
6   2.178955 -1.022584427           17       Easy 0.5567070       1   1_1_14
7   1.614694 -1.588907098           16       Easy 0.5956658       1   1_1_20
8   1.778268 -1.424735539           16       Easy 0.6372482       1    1_1_2
9   2.335790 -0.865176269           22       Easy 0.6372482       1    1_1_1
10  1.612335 -1.591273937           15       Easy 0.6372482       1    1_1_1
11  1.894304 -1.308275088           18       Easy 0.5637444       1   1_1_20
12  2.406307 -0.794402170           28       Easy 0.6372482       1    1_1_2
13  2.604714 -0.595270021           30       Easy 0.6372482       1    1_1_3
16  2.089003 -1.112864992           23       Easy 0.6372482       1    1_1_4
17  2.280552 -0.920615863           24       Easy 0.6372482       1    1_1_5
18  2.894565 -0.304360810           27       Easy 0.6372482       1    1_1_4
19  2.207817 -0.993616599           20       Easy 0.6018322       1    1_1_6
20  3.070550 -0.127732493           29       Easy 0.5869091       1   1_1_13
21  3.072235 -0.126041827           23       Easy 0.5526629       1   1_1_17
22  2.973671 -0.224965511           25       Easy 0.5956658       1   1_1_15
23  2.951776 -0.246940533           23       Easy 0.5966747       1   1_1_12
24  3.110692 -0.087443697           27       Easy 0.5603851       1   1_1_19
28  2.922830 -0.275991968           29       Easy 0.5923149       1    1_1_7
29  1.965266 -1.237053993           22       Easy 0.5526629       1   1_1_16
30  2.238862 -0.962458221           22       Easy 0.6372482       1    1_1_9
31  3.090289 -0.107921270           21       Easy 0.6372482       1   1_1_10
32  2.183489 -1.018034262           25       Easy 0.6372482       1    1_1_5
33  2.588561 -0.611482611           29       Easy 0.5994281       1   1_1_11
34  2.227760 -0.973601010           30       Easy 0.6372482       1   1_1_11
35  3.122679 -0.075412880           31       Easy 0.6372482       1   1_1_12
36  2.407609 -0.793094794           28       Easy 0.5946056       1   1_1_18
37  2.312506 -0.888545545           31       Easy 0.6372482       1    1_1_9
40  3.484956  0.288187249           34       Easy 0.5923149       1   1_1_14
42  5.635369  2.446455755           43       Easy 0.5883785       1    1_1_8
43  3.007182 -0.191332195           37       Easy 0.6372482       1   1_1_10
44  3.376592  0.179427122           42       Easy 0.6039494       1   1_1_15
46  3.491807  0.295063760           37       Easy 0.5883785       1    1_1_6
47  2.459733 -0.740780606           35       Easy 0.5923149       1   1_1_16
49  4.342415  1.148778409           58       Easy 0.5923149       1   1_1_17
50  4.072070  0.877445867           43       Easy 0.5956658       1   1_1_19
27  1.411838 -1.792503276           12     Medium 0.6233724       1   1_2_31
39  2.444012 -0.756559022           12     Medium 0.5656060       1   1_2_10
410 1.384343 -1.820099278           14     Medium 0.5270103       1   1_2_12
52  3.145342 -0.052667395           17     Medium 0.5432581       1   1_2_11
66  2.046874 -1.155147930           19     Medium 0.5300052       1    1_2_5
73  2.597187 -0.602825101           24     Medium 0.5976186       1   1_2_28
81  1.719154 -1.484065521           15     Medium 0.5270103       1   1_2_29
91  2.128899 -1.072822887           24     Medium 0.6124280       1   1_2_32
101 1.723438 -1.479765734           17     Medium 0.5976186       1    1_2_1
111 2.251466 -0.949808963           19     Medium 0.5945631       1    1_2_2
121 3.048752 -0.149610150           18     Medium 0.6007560       1    1_2_3
131 1.824228 -1.378607494           14     Medium 0.6072892       1    1_2_4
14  2.403531 -0.797188095           18     Medium 0.5831041       1   1_2_16
15  2.153502 -1.048130354           19     Medium 0.5633372       1    1_2_5
161 2.986529 -0.212060696           22     Medium 0.6072892       1    1_2_6
171 2.820980 -0.378214307           22     Medium 0.5315473       1   1_2_10
181 2.281344 -0.919821460           24     Medium 0.5777957       1    1_2_1
191 3.008201 -0.190309297           24     Medium 0.6072892       1    1_2_7
201 2.652943 -0.546864982           22     Medium 0.6177849       1   1_2_13
211 2.915123 -0.283727099           21     Medium 0.5777957       1   1_2_28
221 2.679773 -0.519936598           23     Medium 0.6233724       1   1_2_21
231 2.268238 -0.932975426           25     Medium 0.5945631       1    1_2_8
241 1.798613 -1.404315741           19     Medium 0.5589543       1   1_2_25
25  3.308024  0.110608885           33     Medium 0.5589543       1   1_2_17
26  2.270950 -0.930253376           21     Medium 0.5886865       1    1_2_9
281 3.957684  0.762642085           23     Medium 0.6007560       1    1_2_7
291 2.387805 -0.812971513           24     Medium 0.6072892       1    1_2_4
301 3.058489 -0.139837876           26     Medium 0.5270103       1   1_2_26
321 2.711885 -0.487707546           27     Medium 0.5045013       1   1_2_12
331 2.071932 -1.129998849           24     Medium 0.5831041       1   1_2_11
341 3.501225  0.304515946           30     Medium 0.5930652       1    1_2_3
351 2.501880 -0.698480125           32     Medium 0.5397386       1   1_2_15
361 2.829178 -0.369986562           23     Medium 0.5227286       1   1_2_19
371 3.794321  0.598682782           39     Medium 0.5186816       1   1_2_23
38  3.767951  0.572215688           36     Medium 0.5589543       1   1_2_20
401 3.640559  0.444358631           31     Medium 0.5727429       1    1_2_2
41  3.663490  0.467373598           28     Medium 0.6177849       1   1_2_13
431 3.853886  0.658464924           35     Medium 0.5112206       1   1_2_27
441 2.430227 -0.770394389           25     Medium 0.5469294       1   1_2_14
45  2.782266 -0.417069970           34     Medium 0.5227286       1   1_2_14
461 3.288312  0.090825285           34     Medium 0.5727429       1   1_2_15
471 3.501974  0.305267587           34     Medium 0.6007560       1   1_2_16
48  3.031977 -0.166446664           39     Medium 0.5844732       1   1_2_17
491 3.463606  0.266758824           42     Medium 0.5679286       1    1_2_8
501 2.778388 -0.420961929           24     Medium 0.5315473       1   1_2_18
51  3.928336  0.733186576           29     Medium 0.5397386       1   1_2_19
53  3.185791 -0.012070444           34     Medium 0.5148511       1   1_2_22
54  4.883517  1.691857337           28     Medium 0.5886865       1   1_2_20
56  3.589424  0.393036667           30     Medium 0.5589543       1    1_2_9
57  3.766042  0.570299604           36     Medium 0.6007560       1    1_2_6
58  3.794609  0.598971535           30     Medium 0.6007560       1   1_2_21
59  3.519034  0.322390007           37     Medium 0.5315473       1   1_2_22
60  4.524365  1.331392706           44     Medium 0.5363620       1   1_2_23
61  3.936997  0.741879554           41     Medium 0.5432581       1   1_2_30
62  4.241865  1.047861107           48     Medium 0.5727429       1   1_2_24
63  4.052706  0.858011060           44     Medium 0.5363620       1   1_2_24
64  2.804143 -0.395112659           36     Medium 0.5148511       1   1_2_18
65  2.760472 -0.438943312           35     Medium 0.5844732       1   1_2_25
67  3.211382  0.013614050           37     Medium 0.5547667       1   1_2_26
68  3.898996  0.703739700           29     Medium 0.5315473       1   1_2_27
69  4.066681  0.872037327           45     Medium 0.6072892       1   1_2_32
70  3.465358  0.268517331           45     Medium 0.5547667       1   1_2_29
71  3.690402  0.494383867           44     Medium 0.5844732       1   1_2_30
72  2.786653 -0.412666095           43     Medium 0.6177849       1   1_2_31
110 1.698602 -1.504691943           14       Hard 0.4001698       1      2_1
210 2.408737 -0.791963193           18       Hard 0.4663906       1     2_11
55  1.831426 -1.371382512           13       Hard 0.4257874       1     2_12
610 3.266858  0.069292288           26       Hard 0.3860251       1      2_8
74  2.988662 -0.209920087           21       Hard 0.4152211       1      2_7
82  2.639837 -0.560018946           30       Hard 0.3513447       1     2_13
102 2.410446 -0.790248253           23       Hard 0.4126471       1     2_12
112 2.897689 -0.301225127           26       Hard 0.3124676       1      2_2
122 3.691458  0.495443891           34       Hard 0.3860251       1      2_1
132 2.482668 -0.717762058           24       Hard 0.3860251       1      2_3
141 2.544822 -0.655380914           23       Hard 0.3124676       1      2_2
151 4.012675  0.817834431           26       Hard 0.3124676       1      2_5
172 2.790623 -0.408681593           29       Hard 0.3727302       1      2_4
192 4.752598  1.560459528           26       Hard 0.3727302       1      2_4
202 4.186837  0.992631846           26       Hard 0.3124676       1      2_5
212 3.329146  0.131808505           27       Hard 0.3513447       1     2_13
242 3.196244 -0.001578946           39       Hard 0.3826239       1      2_9
251 3.250160  0.052533487           28       Hard 0.3727302       1      2_3
261 4.963846  1.772479511           34       Hard 0.5113574       1     2_10
271 5.306164  2.116047832           42       Hard 0.3013030       1      2_6
282 4.081560  0.886970399           41       Hard 0.3013030       1      2_6
292 5.536857  2.347584124           29       Hard 0.4312367       1      2_7
302 5.335221  2.145210816           39       Hard 0.4001698       1      2_8
311 3.066387 -0.131910927           36       Hard 0.4001698       1      2_9
332 3.969537  0.774538946           50       Hard 0.5060627       1     2_10
342 5.032428  1.841311904           49       Hard 0.4856379       1     2_11

3.4 Final order of sentences (130 sentences in fixed order)

set.seed(20)
easy.length <- sample(nrow(df.lab.items.easy.final))
easy.final <- df.lab.items.easy.final[easy.length,] %>%
  mutate(difficulty = "Easy")
medium.length <- sample(nrow(df.lab.items.medium.final))
medium.final <- df.lab.items.medium.final[medium.length,] %>%
  mutate(difficulty = "Medium")
hard.length <- sample(nrow(df.lab.items.hard.final))
hard.final <- df.lab.items.hard.final[hard.length,] %>%
  mutate(difficulty = "Hard")